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ABSTRACT 

The primary purpose of this monograph is to describe 
how learning curves can be applied to vocational evaluation 
procedures to enhance the reliability and accuracy of performance 
prediction. Particular emphasis is placed on describing two 
techniques known as the "best 20 percent method" and the "Performance 
Analyzer and Enhancer." The latter is a computer software program 
designed for use with inexpensive microcomputers in both vocational 
evaluation and work adjustment services. Together, they represent an 
effective and practical approach to learning curve applications in 
typical vocational evaluation settings^ Three other approaches to 
learning curve use are also described. They range in complexity from 
Tillman's simple notion of constantly readministering a task and 
plotting performance until "peak performance" is achieved, to 
sophisticated learning curve equations. Each of these techniques, 
while having significant limitations, also has much to offer current 
vocational evaluation practice. However, learning curve technology 
should be viewed as a complement to current vocational evaluation 
practices. It provides an additional means for better understanding 
client capabilities and limitations . Continued research and field 
testing, along with an increased emphasis among educators on 
providing training in learning curve technology to professional 
evaluation personnel, will perhaps lead to a wider use of learning 
curves in vocational evaluation programs as well as further 
refinement and improvement of many of the methods and techniques 
presented in this monograph. (KC) 
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INTRODUCTION 



Vocational evaluation is a comprehensive assessment process which uses a 
variety of techniques including psychological assessment, work samples, 
medical, vocational, and educational information, as well as other methods 
and resources to not only assess the existent functional levels of 
individuals with disabilities, but also, in many cases, to make predictions 
about future functional capacity. These predictions are used for a variety 
of purposes. J 

On a macro level, Shalock and Karen (1979), for example, indicated that much 
of the data derived from vocational evaluation services is often used to make 
decisions about a severely disabled individual's feasibility for vocational 
rehabilitation services within the State-Federal vocational rehabilitation 
system. Implicit in this process is the assumption that this data can be 
used to make reasonably accurate predictions about the likelihood that a 
program of vocational rehabilitation services will lead to employment or 
other acceptable outcomes identified by the State-Federal system. This 
assessment of "rehabilitation potential" is often difficult with severely 
disabled individuals, particularly if they have had little or no previous 
work experience or rehabilitation training. To overcome this difficulty, at 
least in part, specialized services such as vocational evaluation have been 
developed and utilized extensively during the last decade. 

On a micro level, data derived from vocational evaluation techniques such as 
work sample testing and job-site-evaluation in particular, is often used to 
make specific predictions about employment potential for a given job or job 
area, as well as the likelihood the client will benefit from related train- 
ing. A client's performance on one or two administrations of a work sample 
may be used to either screen a client out of further consideration for that 
job cluster, or recommend that the client receive technical training for the 
job, or be placed in the job area immediately. 

The performance measures themselves are most often represented by what 
Blakeraore and Coker (1982) described as "static measures." Examples of these 
types of static measures include mean time to complete a task such as a work 
sample, total time, number of pieces produced and so on. 

The essential characteristic of these static measures is that they do not 
effectively take into account changes in an examinee's performance during 
the actual assessment process. Performance scores on individual trials are 
typically lumped together resulting in a mean score which is then compared to 
a norm group or production standard. Rarely is any systematic effort made 
by the evaluator to contrast changes in performance that occur during the 
actual testing itself. As a result, a client may show dramatic gains in 
performance during the latter stages of testing on a given task, but because 
the scores are then combined with initial scores which may be significantly 
lower, the end result is a depressed mean score which may not accurately 
reflect the client's performance capability. In other words, no systematic 
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effort is made to take into account variation in performance during the fixed 
testing period. 

The inherent assumption with this approach is that performance is highly 
stable. A client's performance after a small number, or in many cases only 
one administration of a task, is considered to be a reliable benchmark for 
not only establishing his current capabilities, but also his future perfor- 
mance potential. This assumption has been widely-accepted, at least 
implicitly, within the practice of vocational evaluation despite the fact 
that research in psychology, education, and industry has shown quite clearly 
that on most tasks, and indeed psychomotor tasks in particular, performance 
improves significantly with practice (Bilodeau & Bilodeau, 1961; Blakemore & 
Coker, 1982; Chyatte, 1976; Crossman, 1959; Dunn, 1976; Fitts & Posner, 1967; 
Newell & Rosenbloom, 1981). 

While current definitions of vocational evaluation do not explicitly indicate 
the predictive component of this service (VEWAA, 1977), there should be 
1 i ttl e doubt, as previous authors have ' -ggested, that prediction is an 
important component of vocational evaluation with many clients. Indeed, as 
early as the 1960's, leading vocational evaluation authorities were suggest- 
ing that prediction is an important focus of vocational evaluation. 

Nadolsky (1969) defined vocational evaluation as "a process which attempts to 
assess and predict work behavior primarily through a variety of subject- 
object assessment techniques and procedures," (p. 23). Gellman (1967) noted 
that vocational evaluation is primarily concerned with assessing an 
individual's present level of functioning and making predictions about future 
levels of functioning. More recently, Dunn (1976b) indicated: 

...the predictive use of vocational evaluation underlies much of 
current practice (although) there seems to be considerable con- 
fusion among evaluators and others as to what prediction is all 
about and how it is done. 

(p. 41) 

If one accepts the premise that prediction will continue to play an important 
role in vocational evaluation, then it is important to carefully examine this 
practice. Are the static measures typically used in current practice 
adequate to ensure reliable and accurate predictions about client capabili- 
ties? The research literature is relatively sparse in this regard. However, 
it woul d appear that based on what evidence is available, the current 
emphasis on using static measures as a basis for prediction is uncertain, at 
best. Blakemore and Coker (1982), for example, in conducting research which 
focused on comparing the accuracy of work sample measures versus learning 
curves to predict future performance found: 

...the traditional static work sample measures provided consistent- 
ly worse estimates of the final performance level than did any 
other techniques used in this study. This finding clearly supports 
the need to use learning curves or other indices reflecting 
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learning for prediction purposes rather than the traditional static 
measures such as mean or total score. 

(pp. 35-36) 

Others (Feuerstein, 1979; Schalock & Karen, 1979) have suggested that with 
regard to the assessment of individuals with mental retardation, the focus 
should shift from the current traditional psychometric approach, one which 
does not take into account dynamic changes in performance during the assess- 
ment process, to an "edumetric" approach. The edumetric approach to assess- 
ment would focus on measuring dynamic changes in performance as a result 
of learning which occurs during the assessment process itself. These changes 
in performance would then be taken into account whenever predictions are made 
about future functional capabilities. 

This approach is in direct contrast to the current psychometric orientation 
of vocational evaluation services. This orientation often relies on using 
one or two administrations of a task as a basis for making predictions about 
future performance. And even within these administrations, little effort is 
made to take into account learning and improvements in performance which may 
occur as a result of increased experience with the task. Instead, the 
clients total performance is combined and a mean score is developed and then 
compared to that of a norm group or in some cases an industrial standard - 
once again based on two important assumptions: performance is stable and 
therefore initial performance is a reliable indicator of future practiced 
performance levels; and secondly, the performance of an inexperienced 
worker or examinee can be reliably compared to that of experienced performers 
to determine both short and long-range suitability for employment. Yet, as 
Dunn (1976a) indicated, comparing the performance of inexperienced, un- 
practiced examinees to that of experienced workers can lead to erroneously 
screening the first group out of future employment despite the fact that 
continued practice can lead to significant advances in performance levels to 
the extent that the examinees may eventually achieve competitive performance 
levels if allowed an adequate period for practice and learning. 

Feuerstein (1979) also suggested that reliance on traditional static measures 
of performance, particularly with regard to mentally retarded individuals 
can only result in a tautological process in which a manifest level of 
functioning, already known to be low, is once again demonstrated by poor 
results obtained by the examinee" (p. 89). 

Clearly, it is vitally important that vocational evaluation practices attempt 
to take into account changes in client performance which are a direct result 
of learning and experience with a task. This is particularly true with 
regard to assessment of individuals with severe disabilities who have 
relatively little work experience and often find themselves engaged in work 
sample, situational assessment, and other psychomotor tasks as part of 
the vocational evaluation process. Where dynamic changes in performance are 
evident, it is essential that these changes be considered when making 
predictions about an individual's performance capability. 

Recognizing the important role prediction is likely to contiuje to play in 
vocational evaluation, both in terms of serving as a basis for making 
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decisions about specific occupational outcomes, and training potential, as 
well as feasibility for vocational rehabilitation services through the 
State-Federal system, it is obvious that every effort must be made to ensure 
that the tools and techniques used as a basis for making such predictions be 
as reliable and accurate as possible. One approach that has been shown to be 
a relatively accurate predictor of performance potential is the learning 
curve. Learning curves have been used in industry, education, psychology, 
and other fields for several decades, as a basis for making predictions about 
human performance potential on a variety of tasks. They also appear to hold 
much potential for use within rehabilitation services. 

The primary purpose of this monograph is to describe how learning curves can 
be applied to vocational evaluation procedures to enhance the reliability and 
accuracy of performance prediction. Particular emphasis is placed on de- 
scribing two techniques known as the "best 20% method 11 and the "Performance 
Analyzer and Enhancer." The latter is a computer software program designed 
for use with inexpensive microcomputers in both vocational evaluation and 
work adjustment services. Together, they represent an effective and 
practical approach to learning curve applications in typical vocational 
evaluation settings. Both approaches were developed through research 
conducted at the Research and Training Center at the University of 
Wisconsin-Stout. This monograph is designed to enhance the utilization of 
this research in applied rehabilitation service delivery settings. 

Finally, it should be noted that while learning curves have much to offer the 
practice of vocational evluation, they are certainly no panacea. As will 
become evident in later sections of this monograph, learning curves are not 
without their limitations. Indeed, they are to some extent, a misnomer. 
Learning curves actually indicate changes in performance. These changes may 
or may not be indicative of the extent of learning that has occured. For 
example, an individual may quickly learn a repetitiva task, but because of 
boredom or fatigue, his performance scores do not reflect the amount of 
learning that has taken place. As a result, they should be regarded as 
useful tools which can further enhance existing vocational evaluation 
techniques rather than substitutes for current practices. When used wisely 
by properly trained vocational evaluation personnel, learning curves have 
much to offer in further enhancing the evaluator's expertise and his or her 
ability to successfully evaluate the widely varying capabilities of the 
clients served. 



THE LEARNING CURVE 



A learning curve is basically nothing more than a graphic depiction of 
changes in performance or output during a specified time period. The term 
learning curve is also used to refer to mathenatical equations which describe 
the relationship between practice and performance. Learning curves provide a 
concrete measure of the rate at which an individual or group of individuals 
are learning a task. In their simplest forms, learning curves are generally 
depicted in either one of the forms shown in Figure 1 and Figure 2. 




TRIALS TRIALS 
FIGURE 1 FIGURE 2 



It is important to note that both Figures are based on the same data. 
However, as is readily apparent, one curve slopes downwards while the other 
curve slopes upwards. Both are correct. But in most cases, the upward 
sloping curve with the X (horizontal) and Y (vertical) axes depicted as shown 
is preferred because it is generally regarded as less confusing. People tend 
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to assume that Improvement in performance should be associated with an 
upward swing in a graph. 

Another common way to express a learning curve is to use percentages instead 
of time values for the Y axis. For example, in both Figure 1 and Figure 2, 
time values are expressed on the Y axis with trials expressed on the X axis. 
However, as is indicated in the example provided in Figure 3, these time 
values can be replaced by another measure, the percent of standard. This is 
a useful approach because it indicates quite clearly the extent to which an 
individual 's performance is approaching the production standard or norm. 
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TRIALS 
FIGURE 3 

To convert an actual time score to a percent of standard value, the following 
formula is used: 




(Standard / Time Obtained) X 100?: = t of Standard 



For example, if the industrial standard for a simple work sample assembly 
task is 15 minutes then it is possible to determine the percent of standard 
for each of the time values shown in Figures 1 and 2. Using the formula, 30 
minutes would convert to 50% of the standard and 20 minutes would convert to 
75% of the standard and so on. Thus, it is apparant that the data displayed 
by this type of learning curve would provide an evaluator with a readily 
useful tool for assessing not only changes in a client's performance as a 
result of learning, but also the relationship of the client's performance to 
the standard. 
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Learning curves are useful because they clearly indicate changes in perfor- 
mance based upon learning. Some important characteristics of learning curves 
in general are worth noting. First, performance tends to improve with 
practice. Second, the rate of improvement is often very rapid when someone 
first begins learning a task* This rate, however, tends to slow as the 
amount of practice increases. This is the leveling off phenomenon observed 
in mo*t learning curves. Finally, there is some point beyond which further 
significant increases in performance will not occur. It should also be noted 
that other factors besides learning can influence performance. These include 
factors such as environmental concerns, fatigue, distraction and others. 

Learning curves have been used to not only measure the rate at which an 
individual or group of individuals learn a task, but also to compare that 
rate to an existing standard. Thus, it is not surprising that learning 
curves have been used to a limited extent, in industry, as performance 
appraisal tools (Stevenson, 1982). They have also been used in industry to 
make predictions about long-term improvements in performance that can be 
expected as a result of the curvelinear relationship that exists between 
performance level and the amount of practice an individual has. In other 
words, industry has long-recognized that as workers' gain experience with a 
task, performance can be expected to increase significantly. These increases 
can be taken into account and predicted relatively early on for some tasks, 
resulting in improved production scheduling, better labor cost estimates and 
so on. The same principles can be applied, to some extent, to the use of 
learning curves in education, psychology and vocational evaluation. A brief 
review of how learning curves have been used in industry may be useful in 
better understanding the diverse ways in which they can be used in rehabili- 
tation facilities. 

To date, the primary application of learning curves has been in manufactur- 
ing. They have been used extensively since the 1930' s to help manufacturers 
better plan and schedule work. They have proven to be a useful tool for not 
only better controlling manufacturing costs but also for increasing efficien- 
cy. Learning curves have been developed for specific industrial applications 
where research has shown that a relatively stable relationship exists between 
changes in labor input and unit output. In other words, when first learning 
a task, the more experience individual workers have with a job, particularly 
those that involve psychomotor tasks, the more likely it is that they will 
show significant improvement in performance. This improvement is often quite 
dramatic during the intitial stages of learning and then begins to level off 
and eventually stablize over extended periods of time. 

The Boeing Company is perhaps best known for having recognized the 80% 
learning curve effect for the manufacture of airframes. This effect is based 
on their own research which indicated that for this specific activity, each 
time unit output is doubled, there is a corresponding 20% decline in labor 
input or the time needed to produce the end product. 

For example, if 10,000 hours are needed to produce the first airframe (1 
unit), it can be projected with the use of the 80% learning curve, that only 
8,000 hours will be needed to produce the second unit, 6,400 hours to produce 
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the third unit and so on* Of course, the curve begins to level off with 
poduction reaching a point where improvements in unit output begin to 
stablize. However, the value of such a predictive tool is obvious in terms 
of enhancing manufacturing efficiency. 

An important distinction should be made between different learning curve 
rates. An 80% learning curve, for a given job, indicates that the typical 
worker will show a 20% improvement in performance on that particular job. A 
90% learning curve, for a specific job, indicates that only a 10% improvement 
in performance can be expected from the average worker. A 100% learning 
curve reflects the fact that for that particular job, workers will not show 
any significant improvement in performance even as they gain experience with 
the task. Thus, it is apparent that if one were to draw a learning curve 
representing the tradional static approach to work sample performance 
assessment, an approach which assumes performance is stable and therefore 
will not change with additional trials, a "learning curve" much like the one 
presented in Figure 4 would result. 
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TRIALS 
FIGURE 4 



Tillman (1971) was one of the first proponents to suggest that learning curve 
principles could be applied to vocational evaluation activities in re- 
habilitation. He summarized the problem of using work samples as predictive 
tools in the following manner: 

Most often work samples are used on a single administrative basis. 
This provides information on the client's present level of func- 
tioning. From this information attempts are often made for 
prediction of ability to succeed on a job. There is, however, some 
doubt that knowledge of a client's present level of functioning 
sufficiently indicates his potential for absorbing experience and 
improving. Can we assume, for example, that a client who has had 
no exposure to woodworking hand tools and does poorly on a work 
sample involving these tools has no potential to learn? The answer 
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is no. Would it be feasible to give him a number of trials on the 
task and note any level of improvement on progressive trials? The 
answer is yes. If we plotted improvement on a graph we would 
have a learning curve. 

(P. 1) 

Tillman (1971) went on to provide an example of how a learning curve might be 
plotted for different clients involved in the same task. It is presented in 
Figure 5. 




35 I — i — i 1 1 1 i t i i i 

123 4 56789 10 



TRIALS 
FIGURE 5 

More recently, Coker and Blakemore (1984) used five different types of 
hypothetical learning curves to indicate how this dynamic approach to 
assessment differs from the more traditional static psychometric approach. 
They indicated: 

The advantage of the learning curve approach to evaluating work 
sample performance is that this method reflects what changes occur 
in the client's work sample performance during testing. A static 
process of evaluating the level of functioning, such as using the 
mean or total production rate, fails to account for differential 
performance during the repetitions of the tasks within the work 
sample and the potential for further learning. Individuals 
functioning at the same average level on a work sample involving 
several repeti tions are not necessarily performing comparably 
during the entire session. Figure (6) illustrates this point. 

In Situation A, the ideal ized learning curve is presented where 
skill aquisition increases consistently over time. In Situation B, 
performance deteriorates over time rather than steady improvement. 
It is clear that the client initially performed well, but perfor- 
mance deteriorated; and may have been caused by fatigue, boredom, 
confusion, etc. In Situation C, performance is relatively stable 

9 



9 

ERLC 



except that there is a momentary decline in task performance due to 
distraction, forgetting of instruction, or perhaps lack of parts. 
In Situation D, there is a rapid reaquisition of the task. It is 
indicative of having previously mastered the task and of the 
ability to rapidly return to that level. Finally, steady level 
task performance from first to last repetition is graphed as is 
assumed under traditional work sample administration. Only in 
Situation E is the true current and potential level of task 
performance of 67% of the normative criterion accurate. In 
other situations, current and potential task performance is 
underestimated and additional valuable information about the client 
is lost. It would not be lost, however, if a learning curve 
analysis was used. 

(p. 5) 




FIGURE 6 
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The previous learning curves represent examples of differential performance 
during a work sample with the same normative score of 67%. They may be 
summarized as follows: 

A. Normal acquisition under an idealized learning curve. 

B. Decline in performance due to fatique or lack of motivation. 

C. Acquisition and reacquisi tion due to temporarily forgetting tasks, 
disruption, lack of attention, etc. 

D. Rapid reacquisition to steady state indicating the relearning of a 
previously mastered task. 

E. Idealized practiced worker performance assumed under traditional work 
sample administration. 



Tillman (1971) suggested a method for developing learning curves with 
individuals as well as standardized learning curves for use with groups. He 
indicated: 

If it is desired to establish a standardized learning curve for a 
group, the steps to be taken are as follows: (1) The task must be 
administered to a number of subjects and their time recorded for 
a set number of trials, (2) The average (mean) time taken for each 
trial must be calculated. (3) Each trial's average time is plotted 
on a graph. (4) The final average curve is drawn through the 
points determined by the average curve. The larger the number of 
subjects the smoother the curve will be. 

The number of trials needed to establish a peak of client perfor- 
mance will depend upon the complexity of the task. It would be 
difficult to establish a peak with less than four trials; six to 
ten trials may be needed for more complex tasks. For any given 
work sample, the most practical method of determining the optimum 
number of trials is through the administration of the work sample 
to a number of people. The important point is to give the client 
enough trials to indicate his competency in a given task. 

(P. 2) 



Plotting a learning curve by this method is a relatively simple task. 
However, Tillman's suggestion that clients be allowed to repeat work sample 
performance until supposed "peak performance" is achieved, is a major 
drawback to this approach. Learning and improvement in performance may 
continue almost indefinately on some tasks. Indeed, studies in industry 
(Crossman, 1959; Peterson, 1975) have shown that for some routine, repetitive 
tasks, Tillman, s notion of "peak performance" may not be achieved until 
several thousand trials have been completed. Obviously, such an approach is 
not feasible within vocational evaluation programs. Dunn (1976a) rejected 
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Tillman's approach, suggesting instead that learning curve equations offered 
a viable method for predicting performance potential based on a relatively 
small number of trials. He indicated: 

A possible solution to this problem would be to predict an esti- 
mated practiced performance level of the client from learning curve 
data. Data from a relatively small number of trials could be used 
to establish the equation for the individual's learning curve on a 
work sample. Once this equation has been developed, the vocational 
evaluator could use it to predict the client's estimated perfor- 
mance level after being provided with a certain number of practice 
trials (such as the number of trials provided to new workers by the 
industry for learning) or the number of trials needed for the 
client to reach the established industrial standard or norm. 

(pp. 3-4) 

Dunn conducted research to test this hypothesis with regard to work sample 
testing (1976a). Using data collected by Botterbusch (1974), Dunn used the 
performance scores collected during the first three days of testing to 
predict performance on Day 4, the final day of testing. He found that the 
final performance level could be predicted with less than 1% of error on the 
average. 

The results of this research were very encouraging; however, a major drawback 
to this approach is that it requires evaluators to use relativley sophisti- 
cated mathematical equations to make the predictions. Dunn indicated that 
mathematical skills "at least adequate to develop and solve logarithmic 
equations are required of the evaluator" (p. 11). In addition, this method 
requires substantially more administration time as well as time needed to 
make the actual calculations. These problems obviously tend to limit the 
practical utility of this approach. 

As a result of these problems, Blakemore and Coker (1982) conducted research 
to determine if a simplified, yet accurate and reliable approach to the 
predictive aspect of learning curve use could be developed and applied to 
vocational evaluation activities. 

As part of their research, they studied and compared eight different learning 
curve formulas with regard to accuracy and ease of use in predicting work 
sample performance. The accuracy of each of these methods was also compared 
to the use of traditional static measures used to predict performance 
capability (e.g. mean time, total time, norm reference). Subjects were 
administered a work sample task which consisted of 50 trials per day for five 
consecutive work days. Learning curve equations were applied to the data 
collected during day one of testing, and predictions were made regarding the 
level of performance expected at the conclusion of day five of testing. 

Two major findings resulted from this research. First, all eight learning 
curve methods we re superior performance predictors in comparison to the 
predictive accuracy of traditional work sample measures. Secondly, while the 
learning curve methods varied greatly in terms of their complexity and ease 
of application, the "best-20% method" was as accurate as the other more 
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complex techniques. Because of its ease of use, the best-20% method appears 
to hold significant potential for use in work evaluation settings although 
additional research is needed to confirm the efficacy of this approach. In 
addition, the feasibility of using inexpensive microcomputers to gather data 
and perform the actual learning curve equation calculations was confirmed. 
These findings and the actual processes involved in applying learning curve 
equations and techniques to vocational evaluation practices are discussed in 
the following section. 
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LEARNING CURVE UTILIZATION 



Recognizing that prediction is an important component of vocational evalua- 
tion and that learning curves can provide an effective means for enhancing 
the accuracy of these predictions, it is now important to examine how 
learning curves can actually be applied to the vocational evaluation process 
and related facility operations. In this section, five different approaches 
to learning curve utilization are described along with cautions on each ones 1 
use. 



METHOD NUMBER 1 



Perhaps the simplest approach to learning curve use in vocational evaluation 
and manpower selection in general was suggested by Tillman (1971). As 
briefly indicated in section 2 of this monograph, this approach essentially 
involves providing the examinee or worker with several repetitions of the 
task and plotting the individual's performance on a graph. The end result 
is often a learning curve much like the one indicated in Figure 7. As long 
as the client continues to show improvement, additional administrations may 
take place until Tillman's notion of "peak performance" is achieved. 
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TRIALS 
FIGURE 7 



The main disadvantages of this approach have already been briefly discussed 
in Section 2. First, many hundred or even thousands of trials may be needed 
for certain tasks before the client begins to approach "peak performance." 
Indeed, Cochran (1968) found that punch press operators continued to show 
improvement through 8,000 trials. And Crossman (1959) found that approxi- 
mately four million trials and four years were needed for this group to 
achieve peak performance. 

Secondly, since there is little scientific basis for this approach, it is 
difficult to reliably predict when peak performance has been achieved. For 
example, learning curves typically show a series of improvements followed by 
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plateus in which performance levels off. However, given additional oppor- 
tunity to continue with the task, significant advances in performance may 
continue with new plateus resulting. Because of this, it is difficult for 
the evaluator to reliably predict when the clients Final ultimate level of 
performance has been achieved. 

These two major problems not withstanding, this approach also is limited in 
that it does not necessarily take into account the base performance level 
needed to actually do the job. For example, a client's performance might 
continue to improve as reflected in the learning curve. However, unless the 
evaluator knows the industrial standard or norm for the job, it will be 
difficult for him to reliably determine when this improvement has reached a 
competitive level. 

Industry has provided one limited solution to this problem. Stevenson (1982) 
has suggested that learning curves can be useful in work settings to evaluate 
new workers during training periods. 

This approach can be achieved by recording the new worker's performance in 
much the same manner as suggested by Tillman. However, an addition is made 
to the graph so that the new worker's performance is compared to the typical 
performance improvements shown by other workers who previously performed 
the task. The new worker's performance may also be compared to the standard 
time or production standard developed for the task through industrial 
engineering techniques such as stopwatch time study, predetermined motion 
time systems or other methods. In this way, the new worker's performance can 
be readily compared to the expected rate of learning, which is a dynamic 
measure, rather than the usual static measure such as mean score, alone. It 
can also be compared to the production standard to provide a more concrete 
basis for decision-making about a client's employability. An example of 
this approach is depicted in Figure 8. It indicates that at the end of the 
first week of training the average worker is expected to achieve 75% of the 
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industrial standard (or produce 30 individual units when the standard for a 
fully trained worker is 40 units). As a result, Worker A's performance is 
far below this level and therefore he may be regarded as underqual ified for 
the task. Worker B is viewed as qualified and Worker C is overqual if ied in 
that he far exceeds the normal level of performance. 

While this approach has advantages over that offered by Tillman, it still has 
many of the same limitations. For example, for a disabled client, the rate 
of learning may be slower than the average rate as compared to a nondisabled 
group of new workers. However, this does not necessarily ensure that the 
disabled worker will not be able to reach the standard. Additional training 
time may make it possible for the disabled worker to achieve the standard 
within a reasonable period of additional training. If this factor is 
overlooked, it may result in incorrectly screening clients out of jobs they 
might otherwise be able to perform. In this regard, this approach has many 
of the same limitations of traditional static measures of performance. 

Additional problems with regard to the time needed to develop the worker 
learning curves as well as the reliability of such curves and probable 
shortage of workers on which to develop and test such curves also limits the 
feasibility of this approach. Yet if the evaluation team has the time and 
resources needed to overcome these obstacles, this approach can provide 
useful data for decision-making. It certainly provides valuable information 
not typically offered by the traditional static measures. It might be 
especially useful in job site evaluations where it is possible to develop 
worker learning curves or use existing data provided by the employer. 
Additionally, it would also appear to have potential for use with many work 
sample applications. It could provide learning curve data for comparison 
purposes. 

At the present time, this data is not normally available with either commer- 
cial work sample systems or individual work samples developed independently 
in facilities. However, given the large number of clients who typically 
engage in work sample tasks during the course of a year, and the relative 
ease with which this data can be accumulated and synthesized, it would seem 
feasible to develop this type of learning curve data for specific work 
samples or entire commercial work sample systems. Yet, time constraints, 
costs, and the need for the evaluation staff to develop the learning curves, 
as well as test them on nondisabled as well as disabled individuals alike, 
once again limits their widespread use. Further, it is doubtful whether 
commercial work sample system developers will be willing to make the invest- 
ment needed to develop this data for complete work sample systems, despite 
its usefulness to practitioners. 



METHOD NUMBER 2 

The second major technique involves the application of predetermined learning 
curve rates to client performance. For example, it might be known that a 
certain activity typically shows a learning curve rate of 80%. Another 
activity might typically reflect a learning curve rate of 70% and so on. 
Once these rates have been established for the assigned task, it is then 
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possible to use learning curve formulas and coefficients which can be 
applied to an individual client's performance in order to predict future 
performance levels. 

For example, let's assume that a work sample task is known to have a learning 
curve rate of 80%. Once we have established the client, s initial level of 
performance, we can then estimate future performance using a formula, like 
the one noted below (Stevenson, 1982), along with the data presented in Table 



Tn = Tl x nb 

Tn = Time for the nth unit 
Tl = Time for the first unit 
b = Log learning percent/log2 



Let's assume in this example, that the client is involved with an assembly 
work sample task. As indicated previously, an 80% learning curve rate is 
considered appropriate. In other words, past experience has demonstrated 
that on this task, examinees typically show a 20% improvement in performance 
with practice. Let's assume it took the client 10 minutes to produce the 
first unit. The evaluator would now like to predict how long it will 
take the client to produce the 20th unit, without having to administer the 
task another 19 times. To do this, he works through the following process. 



T20 = 10 minutes ( 20 log .8/log 2 ) 

The evaluator may now use a calculator with a logarithmic function to work 
through the entire formula. However, a more convenient technique is to use 
the predetermined learning curve coefficients offered in Table 1. 

To use the table, the evaluator works through the following process. 

Step 1: Find the desired number of units on which the prediction is to be 
based by simply reading down the unit number column. In this case, the units 
are equal to 20. 

Step 2: Now read across to the appropriate learning curve percentage; in 
this example it is 80%. 

Step 3: Select the applicable UNIT TIME from the appropriate column. In 
this case it is .381 

Step 4: Plug the numbers into the formula noted previously (Tn = Tl X nB) 
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Predicted time 20th unit = 10 minutes(.381) 
Predicted time 20th unit = 3.81 minutes 
Table 1 
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Source: Stevenson, Production/Operations Management pp. 521-522 
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The evaluator can also use the table to predict the total time that will be 
needed to produce all 20 units, based on the time needed to produce the 
initial unit alone. To do this, he works through the following process: 

Step 1: Read down the unit number column and identify the number of units 
to be produced. In this example, 20. 

Step 2: Now read across to the appropriate learning curve percentage; in 
this case it is 80%. 

Step 3: Select the applicable TOTAL TIME from the appropriate column. In 
this case it is 10.485 

Step 4: Plug the numbers into the formula noted below: 

Expected TOTAL TIME for all 20 units = Tl X nB 

Expected TOTAL TIME all 20 units = 10 min. (10.485) 

Expected TOTAL TIME all 20 units = 104.85 minutes 

With this information, the evaluator now has a relatively reliable estimate 
of the client's expected performance level at the end of 20 trials, without 
having to administer the task all twenty times. The advantages of this 
approach are quite obvious and once the predictions have been established, it 
is then possible to compare these performance levels to the applicable norm 
groups or production standards. In this way, the evaluator might observe 
that based on the client's initial level of performance, he was not able to 
meet the production standard for experienced workers; however, through 
application of the learning curve, it is then possible to determine if 
additional trials are likely to result in the client achieving the production 
standard. 

However, it is also evident that there are numerous limitations to this 
approach. First, it is necessary for the evaluation team to establish 
learning curve rates for certain work sample, situational assessment or job 
site evaluation tasks. This would obviously be a very time consuming task 
and it is unlikely that in the near future commercial work sample system 
publishers will provide this data as part of their systems, despite its 
value. With regard to job site evaluation, however, this problem may be 
overcome in the few instances where employers have enough experience with 
the task to have developed their own learning curves. 

A secondary problem of equal importance is the fact that in most cases, the 
established learning curve percentage is based on the learning rate typically 
displayed by average, nondisabled workers. The extent to which this rate 
corresponds with that of disabled examinees in a vocational evaluation 
setting is difficult to estimate. In many cases, it may be possible that 
while an 80% rate for nondisabled workers is appropriate, it may be necessary 
to allow for additional trials for the average client to achieve the expected 
20% gain in performance. Unless this problem is recognized, it could lead to 
errors in predicting client performance. 
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It should also be reemphasized that for a given job, the lower the assigned 
learning curve percentage, the more improvement may be expected from the 
typical worker performing this specific job. For example, a 7056 learning 
curve for a specific job indicates that a 3055 improvement in performance may 
be expected from the average worker, while a 90% learning curve indicates 
that only a 10% improvement in performance should be expected for the average 
worker. A 100% learning curve indicates no expected improvement at all. 
These expected levels of improvement are based on the specific job task and 
cannot be arbitrarily assigned to other tasks. 

Thus it is apparent that while this approach has many advantages over 
conventional static measures, it too has significant limitations in terms of 
practical implementation. If the problem of the time and related resources 
needed to develop learning curve percentages for certain tasks can be 
overcome, it is evident that this approach has many advantages over the 
current practice which primarily relies on the intuitive judgement of the 
evaluator or, in some cases, little more than educated guesses to predict 
future performance levels for clients. 



METHOD NUMBER 3 

This method is somewhat similar to the previously discussed method in that it 
involves using learning curve formulas to predict performance. The major 
difference is that these formulas may be used without the need to have a task 
or job already classified as either having a 70% curve, 80% curve, 90% curve 
and so on. 

Thus it would appear that this method has an important advantage over method 
number two. However, in actual practice, the use of these learning curve 
formulas is limited due mainly to the complexity of most of these formulas 
and the resulting difficulty most evaluators would have in applying them, as 
well as with regard to the time required to manually collect and manipulate 
the data. 

Many different learning curve formulas have been developed. Some are more 
appropriate for certain kinds of tasks than others. Blakenore and Coker 
(1982) for example, studied six different learning curve formulas with regard 
to their predictive use in vocational evaluation. They included: 



x + c 

Y = K ( ) 

X+C + f 

Y = AB X 



Y=A-MB.IogX) V= (LX)/(X-A) 
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Y= — + B 

X 



Y= AX 
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Blaketnore and Coker's research indicated that each of these formulas was a 
more accurate predictor of performance capability than were traditional 
static measures such as mean score. The following example illustrates the 
use of one relatively simple learning curve equation for a client involved in 
a work sample task. 



CALCULATING A LEARNING CURVE EQUATION 

The instructions below describe how to calculate the hyperbolic learning 
curve equation (Y=LX/X-A) developed by Thurstone (1919) using the least- 
squares method described by Barlow (1928). The data used in the example 
were taken from the research study conducted by Blakemore and Coker (1982). 
The data represent a client's performance on the first 25 trials of a work 
sample task. The client eventually completed 250 trials on the task, 50 
trials per day for 5 consecutive work days. The original scores were 
response times but they have been converted to a percentage of (industrial) 
standard measure. 

Although the method described below looks somewhat complex, the calculations 
involved are really quite simple and can easily be done with a hand held 
calculator. The process can become quite time consuming, however, with a 
large amount of data (more than 25 scores or so). For that reason, we 
recommend the use of a microcomputer to perform the calculations if one is 
available. This can simplify the task and speed it up greatly. 

To calculate the learning curve equation, you need to determine the values 
for L and A in the equation Y = (LX)/(X - A). These values, sometimes 
called parameters, are calulated using equations 1 and 2 below: 

L = EY2(EX2Y) - (EXY)(EXY2) * Equation 1 

EY2(EX2) - (EXY)2 

A = EX2Y - L(EX2) Equation 2 

EXY 

The first step in calculating the learning curve is to enter the data into a 
table such as Table 2 and to calculate the values in the table. (Note that 
we have Included a blank table in the appendix for you to use with your own 
data. Simply photocopy that form whenever you wish to calculate a learning 
curve). As you can see from examining Table 2, there are 7 columns. 
The values in the first 2 columns, labeled X and Y, you already have after 
you have collected some data. The X values represent the trial numbers 
(starting at 0 - that is, the first trial is labeled G, the second is 1, 
etc.). The Y column contains the scores the individual obtained on each 
trial. The remaining values in the table, columns 3-7, are calculated 
using the values in columns 1 and 2. 

* The E in the formula represents a summing procedure, e.g., 2+2+5 - 9, and 
the 2 represents a squaring procedure, e.g., 2x2 = 4. 
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TABLE 2 



Table to be Used When Calculating the 
Hyperbolic Learning Curve Equation (Y = (L x X)/(X - A)) 



Values Used in the Calculation 
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Follow the instructions below to calculate the values in the table and to use 
the results of those calculations to compute the learning curve equation. We 
assume to begin with that you will already have created (or copied) the table 
and have entered the X and Y values and added the Y values. 

Step 1. In the column labeled X2 (Column 3) put the square (the number 
times itself) of each X value (found in Column 1). Then add the X2 values 
and place the sum at the bottom of the column. 

Step 2. In the column labeled Y2 (Column 4) put the square of each Y 
value (Column 2), add all of the Y2 values, and place the sum at the bottom 
of the column. 



For example: 



Y Y2 

39 1521 

41 1681, etc, 



Step 3. Multiply each X value by its corresponding Y value and place the 
results in the XY (for X times Y) column (# 5). Then add all of these values 
and place the sum at the bottom of the column. 



For example: 



X....Y XY 

0 39 0 

1 .. 41 41 

2 .. 52 104, 



etc. 



Step 4. Take each X score and multiply it by its corresponding Y2 (Column 
4) value. Add all of these XY2 values and place their sum at the bottom of 
the column. 



For example: 



X .. 


. Y2 .... 


.. XY2 


0 .. 


1521 ... 


, , 0 


1 .. 


1681 


.. 1681 


2 .. 


2704 .. 


.. 5408, etc. 



Step 5. Take each X2 value (Column 3) and multiply it by its correspond- 
ing Y value (Column 2) and place the result in the X2Y column. Add the X2Y 
values and place the sum at the bottom of the column. 



For example: 

X2 ... Y X2Y 

0 ... 39 0 

1 ... 41 41 

4 ... 52 208, etc. 



You have now completed the table and are ready to use the values in the table 
to calculate the values of L and A (Equations 1 and 2). 
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Step 6. Take the sum of the Y2 values (Column 4) and multiply it by the 
sum of the X2Y values (Column 7). 

106,651 x 355,178 = 37,880,088,800 

Step 7. Take the sum of the XY values (Column 5) and multiply it by the 
sum of the XY2 values (Column 6). 

21,064 x 1,501,068 = 31,618,496,350 

Step 8. Take the value obtained in Step 6 and subtract the value obtained 
in Step 7 from it. This is the numerator (top part) of Equation 1. 

37,880,088,800 - 31,618,496,350 = 6,261,592,450 

Step 9. Take the sum of the Y2 values (Column 4) and multiply it by the 
sum of the X2 values (Column 3). 

106,651 x 4,900 = 522,589,900 

Step 10. Take the sum of the XY values (Column 5) and square it. 

21,064 x 21,064 = 443,692,096 

Step 11. Take the value obtained in Step 9 and subtract the value 
obtained in Step 10 from it. This is the denominator (bottom part) of 
Equation 1. 

522,589,900 - 443,692,096 = 78,897,804 

Step 12. Divide the value obtained in Step 8 by the value obtained in 
Step 11. This is the value of the L parameter in the learning curve equa- 
tion. 

6,261,592,450/78,897,804 = 79.36 
Step 13. Multiply the value of L times the sum of the X2 values (Column 

3). 

79.36 x 4900 = 388880 



Step 14. Subtract the value obtained in Step 13 from the sum of the X2Y 
(Column 7) scores. This is the numerator in Equation 2. 

355,178 - 388,880 = -33702 

Step 15. Divide the value obtained in Step 14 by the sum of the XY (Column 
5) scores. This gives you the value of the A parameter. 

-33702/21,064 = -1.599 
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You have now completed the computation of the L and A parameters in the 
hyperbolic learning curve equation. You can now use these values to compute 
the Y value predicted using the learning curve equation for any value of X 
(i.e., any amount of practice). To do this you simply plug the X, L, and A 
values into the equation and compute the resulting Y value. For example, 
using the values we arrived at in the example given above we can make a 
prediction of where performance would be after 30, 40, and 50 trials of 
practice at the task. These would be calculated as shown in A, B, and C 
below: 

A. The predicted Y value after 30 trials would be 

Y = (79.36 x 30)/(30 - -1.599) = 

2380.8 / 31.599 = 75.34 

B. The predicted Y value after 40 trials wpuld be 

Y = (79.36 x 40)/(40 - -1.599) = 

3174.4 /41.599 = 76.31 

C. The predicted Y value after 50 trials would be 

Y = (79.36 x 50) / (50 - -1.599) = 

3968 / 51.599 = 76.90 



These calculations indicate that the individual who was being tested on this 
work sample could, based upon performance during the first 25 trials of the 
task, be expected to perform at about 75% of standard during trials 30, 40, 
and 50. The individual could also be expected to still be showing some 
improvement on the task, as well. 

As mentioned above, the data used in this example represent an individual's 
actual performance on the initial 25 trials of a work sample task. As part 
of the research that Blakemore and Coker (1982) conducted, tUe data from the 
initial day on the task was used to predict the average performance level 
attained on the last day of practice (Day 5). The prediction was then 
compared with the actual level of performance that was attained on Day 5. 
The same was done with the data that was analyzed above. Using the parameter 
values (L = 79.36 & A = -1.599) from above, the predicted average score for 
Day 5 (Trials 201 - 250) is 79% which is 12% lower than the actual average 
score {91%) for those trials. The difference between the predicted and the 
attained scores represents prediction error. This difference is considerably 
less than that which would have been obtained used the average score (64%) 
for the first 25 trials been used, however. Thus, the benefit of using 
learning curves to make predictions can be seen. 
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It is important to note that while the results of the research conducted by 
Blakemore and Coker (1982) clearly indicated that each of the learning curve 
formulas they tested was more accurate in predicting eventual performance 
levels than were traditional static measures used with work samples, the 
authors noted the difficulty the average evaluator is likely to have in 
readily applying these formulas because of the relative compexity of the 
mathematics involved as well as the time needed to apply the formulas. In 
addition, it must also be noted chat any learning curve formula tends to lose 
accuracy and reliability the further into the future the evaluator attempts 
to predict. For example, while it might be realistic for an evaluator to 
attempt to predict a client's performance at the twentieth trial or after 10 
days of job training, based on a relatively small number of initial trials, 
it would be unrealistic to try and predict performance on the 1,000th trial 
or after 6 months of training, based on these formulas alone. Some 
researchers in this area (Blakemore & Coker, 1982 ; Trussell, 1966) have 
suggested that a general guideline to be followed is that it is only safe to 
predict approximately five times as many trials into the future as the number 
of trials on which the data is based. Thus, i f. predictions are to be based 
on data gathered from 10 trials, they should not go beyond the client's 
fiftieth trial. Beyond this factor, the reliability and accuracy of the 
predictions can decline substantially depending on the tasks involved. 

Another important point for evaluators to keep in mind when using these 
formulas is that it is generally advantageous to administer the task during 
several discrete sessions, separated by rest periods for the examinee. This 
will result in the accumulation of data which will, in most cases, better 
reflect the client's potential for improvement. In thic way, problems of 
fatigue, distractabil ity , and outright boredom with the task can be better 
managed. If a learning curve is based on one continuous sitting over a 
period of several hours, obviously the client is more likely to become 
fatigued and this will generally depress performance. Using more sessions, 
with breaks in between, often results in a learning curve like the one noted 
in Figure 9. The fact that people tend to show significant performance 
improvements and then level out followed by additional improvements in 
the next session, with a new stabalizing level established and so on indi- 
cates the advantage of this approach in terms of obtaining more accurate data 
on cl ient learning. 

Finally, it should also be recognized that with regard to the use of any of 
these formulas, the more trials the client actually engages in, the more data 
can be accumulated, and the accuracy of the learning curve projections are 
then enhanced. However, this can have practical limitations. For example, 
administration of a single work sample or situational assessment task which 
requires several hours to complete obviously limits the number of times it 
will be repeated and the resulting improvements in performance that can be 
plotted. On the other hand, some tasks that require only a few minutes to 
perform might be repeated several dozen times, thereby providing a wealth of 
data. Even in this case, however, there is a tradeoff. For as the example 
in this section suggested, the more data that is accumulated, the more time 
that will be required to compute all the data. Thus time considerations 
must be considered in relation to not only the number of trials, but also the 
additional time that will be needed to collect and analyze the resulting 
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40- 




TRIALS 
FIGURE 9 

wealth of data. One practical solution to this problem is the use of 
microcomputers to process the data in such cases. This method is described 
in method 5 discussed later in this section. It provides a viable alterna- 
tive to gathering large amounts of data reflecting client learning, without 
overwhelming evaluators with manually computing all of the data and 
translating it into a learning curve. 

It should also noted that the example presented in this section represents 
one of the simplest learning curve formulas available. Other formulas are 
much more complex, require more data, and also more mathematical training 
among users. This problem tends to severely limit their use in most evalua- 
tion settings. Dunn (1976a) for example pointed out that evaluators 
would need, in most cases, mathematical skills that would at least, as a 
minimum, allow them to sol ve logarithmic equations. These ski lis are not 
normally part of a vocational evaluator's training at the present time. One 
solution to this problem i^, however, as suggested earlier, to use micro- 
computers to process much of the data. 



28 



32 



Of perhaps more significance, Blakemore and Coker (1982) found that another 
technique termed the "best-20% method" was as accurate a predictor as any of 
the learning curve formulas mentioned, and equally importantly, it was 
relatively easy for evaluators to use. The researchers indicated: 

.••if one were to make a recommendation about which prediction 
method to use, based upon the present findings, the best-20% method 
would probably be the most reasonable choice. This method was 
found to be as accurate as any of the learning curves yet is easier 
to compute and requires data from only one practice session. This 
conclusion should be tempered, however, by the possibility that 
future research might demonstrate deficiencies in the accuracy of 
the best-20% method, 

(p. 32) 

In the following section, the best-20% method is described and an example of 
how to use it is presented. 



METHOD NUMBER 4 

The so-called "best-20% method" was developed and described by Blakemore and 
Coker (1982) with regard to their work on learning curve use in vocational 
evaluation. As the researchers indicated, this method was developed in an 
effort to find a "practical yet accurate prediction technique that was based 
on the well-demonstrated fact that performance improves with practice" 
(p, 9), It was described by the researchers as follows: 

this method consisted of using the mean of the fastest 20% of the 
trials during the first practice session as the estimate of the 
individual's final performance level, 

(P. 9) 

For example, in the case of Blakemore and Coker's (1982) work, the examinee 
was administered a single work sample task each day for for five consecutive 
work days. Each one of the administrations consisted of 50 trials. Using 
the best-20% method then, the average score for the fastest ten trials, 
during the first day of administration, was calculated. This score repre- 
sented the predicted performance level which the client would achieve by 
the end of the fifth day of work on the task; or, in other words, after 
another 200 trials. 

Suppose a client was involved in a simple assemby work sample. One admini- 
stration of the work sample involved assembling 50 units and recording the 
time score for each unit assembled. The following data depicted in Table 3 
might then result. 
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TABLE 3 



TRIAL 


RAW SCORE 
(in seconds) 


RANK 


TRIAL 


RAW SCORE 
(in seconds) 


RANK 


1. 


27.5 




26. 


15.8 




2. 


26.4 




27. 


20.1 




3. 


16.7 




28. 


14.0 


8 


4. 


16.8 




29. 


23.2 




5. 


15.2 




30. 


16.5 




6. 


20.6 




31. 


13.3 


4 


7. 


21.3 




32. 


32.5 




8. 


16.7 




33. 


22.3 




9. 


14.8 




34. 


14.2 


9 


10. 


41.9 




35. 


15.8 




11. 


16.0 




36. 


22.6 




12. 


16.6 




37. 


13.3 


4 


13. 


14.8 




38. 


16.5 




14. 


19.4 




39. 


14.7 




15. 


15.4 




40. 


15.7 




16. 


16.2 




41. 


15.7 




17. 


18.4 


# 

3 


42. 


16.5 




18. 


13.2 


43. 


16.5 




19. 


12.6 


2 


44. 


14.8 




20. 


26.4 




45. 


14.7 




21. 


17.2 




46. 


18.5 




22. 


24.2 




47. 


13.9 


7 


23. 


20.7 




48. 


17.2 




24. 


14.5 


10 


49. 


11.7 


1 


25. 


13.7 


6 


50. 


15.1 





Using the best-20% method, the evaluator would now take the 10 fastest scores 
and determine the mean. In this case the ten fastest scores in seconds 
are: 11.7, 12.6, 13.2, 13.3, 13.3, 13.7, 13.9, 14.0, 14.2, 14.5. 

To find the mean, or the average, the evaluator now simply adds each of the 
time values and divides the total amount by 10. The result is a mean value 
of 13.44 seconds. This would then be considered the predicted performance 
level which the client could be expected to achieve given additional trials 
and opportunity to learn the task. 

In some cases it may be necessary to modify this approach somewhat. For 
example, in this research, a single administration of the work sample 
actually resulted in 50 trials for the client and fifty time values. This 
provided an adequate amount of data for making the predictions. However, in 
some cases, a work sample or situational assessment task may result in 
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obtaining only one score for the entire task during each administration. 
And the actual standard time needed to complete the task may take several 
minutes or even a few hours. In such cases, the evaluator obviously will not 
be able to collect as many data points as in the research example. However, 
it is recommended that the task be administered as many times as possible 
since this will help ensure better accuracy in prediction. In most cases, at 
least ten pieces of data or time values should be obtained for a task before 
attempting to use the best-20% method. 

Clearly, the best-20% method offers a number of advantages over other methods 
previously described. Its ease of use makes it attractive for application in 
vocational evaluation settings. And obviously, the research indicating its 
accuracy is also a strong reason for using this method. However, like the 
other methods described, it too has its limitations. 

First, as was suggested previously, some evaluation tasks will not lend 
themselves to this method. This is true with regard to many situational 
assessment tasks, job-site evaluations, or work sample tasks which require 
extended periods of time to complete with only one time value provided at the 
end of each administration. For example, if a client is assigned to a 
janitorial task which requires approximately three hours for the average 
examinee to complete, practical considerations 1 imit the number of times 
the task can be readministered. And unless an adequate number of time values 
are obtained, clearly the best-20% value will have little reliability. 

Another possible drawback to this approach was suggested by Blakemore and 
Coker (1982). They indicated that while this approach appeared to offer 
accurate predictions in comparison to traditional learning curve formulas, 
more research is needed to support their findings since relatively little 
work has been done in this area. They specifically indicated more research 
is needed to further test the overall reliability and accuracy of this 
method, as well as whether or not the 20% factor is the best possible 
value. Further research might indicate that a different percentage factor 
might offer a more accurate basis for prediction. 

Despite these drawbacks, the best-20% method has much to offer in terms of 
accuracy and ease of use, particularly in comparison to current static 
prediction methods. As Blakemeore and Coker (1982) indicated, "the best-20% 
method can certainly reduce the amount of prediction error when compared to 
the use of the traditional static performance measure of work sample perform- 
ance" (p. 33) . 



METHOD NUMBER 5 

It should be apparent from reviewing each of the methods discussed thus far 
that learning curves hold much potential for use in vocational evaluation. 
In addition, learning curve formulas and the best-20% method offer proven 
techniques for ensuring accuracy in these predictions. However, both 
approaches also require that a relativley large amount of data be collected 
on client performance. The more data collected, the better predictive 
accuracy is enhanced. However, collecting the data is often time consuming 
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and attempting to manually analyze it can be an overwhelming task for 
evaluators with limited mathematical training. This process is also prone to 
error. Thus, there is clearly an important tradeoff. These problems are even 
more pronounced when evaluators attempt to use sophisticated learning curve 
equations as described in method number 3. As a result, to date, the use of 
learning curves has been limited by practical considerations. 

One way to overcome these obstacles is to use relatively inexpensive yet 
powerful microcomputer technology to not only assist the evaluator in 
collecting the data but also in making the mathematical computations that are 
involved in applying the learning curve formulas. The "Performance Analyzer 
and Enhancer" is a computer software program designed to meet just these 
needs. It runs on a relatively low cost 64K Commodore microcomputer. The 
primary functions of the Performance Analyzer and Enhancer are to: 

1. Collect data relating to the amount of time it takes an individual 
to perform a repetitive task. 

2. Analyze the data collected. 

3. Calculate learning curves and make predictions about the level of 
performance the individual can be expected to achieve with additional 
learning and experience. 

4. Increase or enhance the individual's level of performance by pro- 
viding feedback following each repetition of the task, to the 
examinee. 

A photo of the Performance Analyzer and Enhancer configuration for a work 
sample in Appendix B. 

During administration of a work sample task, the microcomputer is basically 
linked to a remote switch which is attached to the work sample and thereby 
automatically indicates when the work sample or task has been completed by 
the client. The time needed for the client to complete the task is recorded 
by means of the switch which automatically starts and stops the timer. One 
of the primary advantages of this system is that the switching can be hooked 
up in a variety of different ways so that the software and the computer 
itself can be used with a variety of different kinds of existing work samples 
or job samples, particularly those that are repetitive in nature and involve 
psychomotor tasks. 

The Performance Analyzer and Enhancer is a versatile software program which 
is designed to collect substantial amounts of data on client performance and 
plot lea mi ng curves which can be used for predictive purposes. Three 
different learning curves may be plotted based on each of the three 
equations. In addition, the best-20% method may also be calculated. Users 
may choose to plot curves using all four methods since some variation can be 
expected. Once this variation is established, it can be taken into account 
when making decisions about the most optimal performance level expected and 
the least optimal perrformance level expected. In addition, calculating all 
four methods also helps offset the fact that one technique or equation may be 
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more appropriate and more accurate than other techniques, for a given 
task. In most cases, the end result of this effort is that a printout 
similar to the one shown in Appendix C is provided. 

Clearly, this type of information is very useful to evaluators and clients in 
terms of gaining a more accurate picture of improvements in a client's 
performance as a result of learning. And the predictive use of the data is 
equally important. Further, the Performance Analyzer and Enhancer, as a 
computer based fystem, is able to overcome many of the problems associated 
with the other methods described in this section, chief of which are the time 
needed to compute the learning curves and analyze the data if a manual system 
is used. 

It should be noted that the Performance Analyzer and Enhancer has a number of 
other uses beyond those related to learning curve applications. It can be 
used in work adjustment programs to enhance client performance by providing 
clients with visual and auditory feedback while they work. Clients may 
observe a video monitor showing colorful graphics designed to display their 
performance and compare it to some desired goal. Thus the Performance 
Analyzer and Enhancer represents more than simply a feasible solution to the 
need to use learning curves in vocational evalaution. 

Finally, it must be noted that the Performance Analyzer and Enhancer is still 
in the experimental stage. Initial field tests indicate much promise for 
this software; however, additional testing and research will be conducted 
throughout 1985 and possibly into 1986. It is not expected that this tool 
will be available for widespread commercial use in facility programs until 
each phase of the testing has been completed. 
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SUMMARY 



Learning curves have long held much potential for use in vocational evalua- 
tion. However, to date, their use has been limited because of three im- 
portant factors: (1) vocational evaluation personnel have had relatively 
little training in the use of learning curves, particularly with regard to 
the application of learning curve equations; (2) manually collecting and 
computing the data that makes up most learning curves is a time-consuming 
process for evaluators who are already constrained for time in many cases; 
and (3) perhaps evaluators, educators, and to some extent referral sources 
are guilty of complacency in continuing to accept the predictive validity of 
current static approaches to assessment despite the glaring paucity of 
research or other information supporting this notion of "presumptive 
val idity." 

Yet, despite its value, it should be equally apparent that learning curve 
technology is not without its limitations. The use of learning curve 
equations can be a complicated, time consuming process unless microcomputer 
technology is used. Current efforts to integrate microcomputer technology 
into this process hold much promise but much work still needs to be done in 
this area before a viable system is available on a widespread basis. One 
possible alternative to this approach is the best-20% method. It provides a 
practical and easy-to-use approach to learning curve analysis. Yet, even 
here, additional research is needed to validate this approach. And while the 
simplest approach, involving little more than administering a task several 
times and plotting changes in performance, is useful in gaining more under- 
standing of client capabilities than current static procedures offer, the 
problems of objectively determining when "peak performance" has been 
achieved, time needed for several administrations and related problems come 
to the forefront once again. 

Thus, it is clear that learning curves are not a panacea. Use of any of the 
five methods described in this research utilization report will undoubtedly 
add valuable information to one's understanding of a client's capabilities 
when prediction is a significant goal of the evaluation. It must be recog- 
nized, however, that there are boundaries to these predictions. Use of 
even the most sophisticated learning curve equations, based on hundreds of 
trials, still does not allow the evaluator to make absolute predictions about 
"ultimate" performance capability. There is a limit to the predictive 
capacity of learning curves and depending on the approach used, number of 
trials available etc., this limit can be reached fairly quickly. One cannot 
provide a client with a few trials and then predict into the infinite future 
what absolute performance level is possible. Predictions must be tempered by 
the reality of the evaluation setting itself and limitations of the tech- 
nology. Thus, while it is possible to predict with relative accuracy how a 
client will perform on the fiftieth trial based on only ten trials, evalua- 
tors must recognize that as predictions extend further into the future, their 
rel iabil ity and accuracy can become increasingly doubtful . With this in 
mind, it should be apparent that learning curves offer much in terms of 
complementing many existing existing vocational evluation practices. They 
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should not, however, be viewed as a substitution or replacement for existing 
practices. 

This monograph has presented a rationale for the integration of learning 
curve technology into current vocational evaluation practice. Five different 
approaches to learning curve utilization have also been described. They 
range in complexity from Tillman's simple notion of constantly readminister- 
ing a task and plotting performance until "peak performance" is achieved, to 
sophisticated learning curve equations and the use of the microcomputer-based 
Performance Analyzer and Enhancer. 

Each of these techniques, while having significant limitations, also has much 
to offer current vocational evaluation practice. However, learning curve 
technology should be viewed as a complement to current vocational evaluation 
practices. It provides an additional means for better understanding client 
capabilities and limitations. Continued research and field-testing, along 
with an increased emphasis among educators in providing training in .learning 
curve technology to professional evaluation personnel, will perhaps lead to a 
wider use of learning curves in vocational evluation programs as well as 
further refinement and improvement of many of the methods and techniques 
presented in this monograph. 
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Table to be Used When Calculating the 
Hyperbolic Learning Curve Equation (Y = (LX )/ (X - A)) 



Values Used in the Calculation 



X Y X2 Y2 XY XY2 X2Y 





0 
1 
2 
3 
4 
5 
6 
7 
8 
9 

10 
11 
12 
13 
14 
15 
16 
17 
18 
19 
20 
21 
22 
23 
24 
25 














Total 












1 
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NAME: DflN 

DATE- 3/.V84 SESSION NUMBER : 1 

TIME: 8 AM 

TASK: MAIL SORT 



NUMBER OF PIECES/ASSEMBLY: 25 
PIECES/ASSEMBLIES PER TRIAL : 1 
NUMBER OF TRIALS: 25 
PREVIOUS TRIALS COMPLETED : 8 



NORMATIVE BASE LINE 

TIME HOURS : 8 

MINUTES : 1 
SECONDS : 0 

PER 1 PIECES/ASSEMBLIES 



STANDARD TIME FOR ONE TRIAL ■ 6.8 SECONDS 
STANDARD TIME FOR ONE PIECE/ ASSEMBLY = 68 SECONDS 



PERFORMANCE 

MEAN TIME PER TRIAL: 169.3? SECONDS 

MEAN TIME PER PIECE/ASSEMBLY: 169.87 SECONDS 

STANDARD DEVIATION: 72.38 SECONDS 

MEAN PERCENT OF STANDARD : 35.32 K 



QUALITY 

INCORRECT ASSEMBLIES : 0 

CORRECT MEAN TIME PER TRIAL 169.8? SECONDS 

CORRECT MEAN TIME PER PIECE/ASSEMBLV = 169. 37 SECONDS 

CORRECT MEAN PERCENT OF STANDARD : 35.32 K 



SUMMARY 

TOTAL TIME- 4246.86 SECONDS 
FASTEST TRIAL: 88.08 SECONDS 
SLOWEST TRIAL: 327.01 SECONDS 

MEAN OF THE FASTEST 2QK OFTHE TRIALS'- 180.25 SECONDS 
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BAR GRAPH 

1 1 1 

5 Q 5 6 

TRIALS TIME 7. O 9 0 0 0 COMMENT 



2 311.91 13 an AB 

3 310.31 13 BBSS CDE 

4 227.37 26 DBSBS F 

5 327 is mm GH 

6 281.35 23 UHBBBB I 

3 173. 5 33 —MB 

3 S3. 07 63 3MH 

li3 129.33 43 'MU l lill Hf 

12 199! 63 53 M WBW a l< 

14 11Q.33 54 OHKI 

15 120. 63 43 WIIIWMiUH 

is 109.73 53 mmmsB^m 

13 213.43 27 SJ^Lg H 

IP". 7 . 23 33 SgaBBBB 

22 141.55 42 IMWIHiHI 

23 193.33 54 HIBBIlBttW Ha 

24 137.37 43 UWIi 

25 115.1 52 SMailWiffiiM 
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NAME DAN 

DATE: 3/7/84 SESSION NUMBER : 2 

TIME'- 1 PM 

TASK ; MAIL SORT 



NUMBER OF PIECES/ASSEMBLY = 25 
PIECES/ASSEMBLIES PER TRIAL : 1 
NUMBER OF TRIALS : 25 
PREVIOUS TRIALS COMPLETED : 25 



NORMATIVE BASE LINE 

TIME HOURS : @ 

MINUTES : 1 
SECONDS : 8 

PER 1 PIECES/ASSEMBLIES 



STANDARD TIME FOR ONE TRIAL: 68 SECONDS 
STANDARD TIME FOR ONE PIECE/ASSENBLV = 60 SECONDS 



PERFORMANCE 

MEAN TIME PER TRIAL: 35.63 SECONDS 

MEAN TIME PER PIECE/ASSEMBLY: 95.63 SECONDS 

STANDARD DEVIATION: 15.3? SECONDS 

MEAN PERCENT OF STANDARD : 62.74 }i 



QUALITY 

INCORRECT ASSEMBLIES: 1.06 

CORRECT MEAN TIME PER TRIAL: 99.68 SECONDS 

CORRECT MEAN TIME PER PIECE/ASSEMBLV : 99.68 SECONDS 

CORRECT MEAN PERCENT OF STANDARD 1 60.18 % 



SUMMARY 

TOTAL TIME: 2390.75 SECONDS 
FASTEST TRIAL 63.75 SECONDS 
SLOWEST TRIAL: 137.66 SECONDS 

MEAN OF THE FASTEST 20* OFTHE TRIALS : 77.56 SECONDS 
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NAME BAN 

DATE J.T/U4 SESSION NUMBER : 3 

TIME: :j PM 

TASK MAIL SORT 



NUMBER OF PIECES/ASSEMBLY: 25 
PIECES. 'ASSEMBLIES PER TRIAL : 1 
NUMBER OF TRIALS : 25 
PREVIOUS TRIALS COMPLETED ■ 59 



NORMATIVE BASE LINE 

r I ME hours; 9 

MINUTES: 1 
SECONDS : @ 

PER 1 PIECES.-" ASSEMBLIES 



STANDARD TIME FOR ONE TRIAL: 66 SECONDS 
STANDARD TIME FOR ONE PIECE/ ASSEMBLY = 60 SECONDS 



PERFORMANCE 

MEAN TIME PER TRIAL : 94.29 SECONDS 

MEAN TIME PER PIECE/ASSEMBLY • 94.29 SECONDS 

STANDARD DEVIATION : 21.92 SECONDS 

MEAN PERCENT OF STANDARD : 63.62 K 



QUALITV 

IHLORREUT ASSEMBLIES- O 

CORRECT MEAN TIME PER TRIAL : 94.29 SECONDS 

CORRECT MEAN TIME PER PIECE/ASSEMBLV = 94.29 SECONDS 

CORRECT MEAN PERCENT OF STANDARD : 63.62 % 



SUMMARV 

TOTAL TIME: 2357.46 SECONDS 
FASTEST TRIAL- 72. S SECONDS 
SLOWEST TRIAL: 1S4. 69 SECONDS 

MEAN OF THE FASTEST 2SK OFTHE TRIALS- 75.29 SECONDS 
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NAME DAN 

BATE : 3/8/84 SESSION NUMBER '■ 4 

TIME: 10 RM 

TASK- MAIL SORT 



NUMBER OF PIECES/ ASSEMBLY : 25 
PIECES/ ASSEMBLIES PER TRIAL- 1 
NUMBER OF TRIALS : 25 
PREVIOUS TRIALS COMPLETED : 75 



NORMATIVE BASE LINE 

TIME HOURS: 8 

MINUTES: 1 
SECONDS : 9 

PER 1 PIECES/ASSEMBLIES 



STANDARD TIME FOR ONE TRIAL: 68 SECONDS 
STANDARD TIME FOR ONE PIECE/ASSEMBLV : 68 SECONDS 



PERFORMANCE 

MEAN TIME FER TRIAL: 74.34 SECONDS 

MEAN TIME PER PIECE/ASSEMBLV : 74.34 SECOND:; 

STANDARD DEVIATION: 3.23 SECONDS' 

MEAN PERCENT OF STANDARD : SO. 7 X 



QUALITY 

INCORRECT ASSEMBLIES: 8 

CORRECT MEAN TIME PER TRIAL: 74.34 SECONDS 

CORRECT MEAN TIME PER PIECE/ASSEMBLV • 74.34 SECONDS 

CORRECT MEAN PERCENT OF STANDARD : SO. 7 Ji 



SUMMARY 

TOTAL TIME 1858.62 SECONDS 
FASTEST TRIAL: 53.38 SECONDS 
SLOWEST TRIAL"- 93.63 SECONDS 

MEAN OF THE FASTEST 22TA OFTHE TRIALS : 63.44 SECONDS 
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